Skip to content

Conversation

@ngholiza
Copy link
Collaborator

Summary

Enforces experiment data isolation in Postgres with row-level security (RLS), separates the API's database roles, and fixes the experiment-creation and proposal-generation issues found in review. This code is already running in production (dt-api:rls-b02e9ac); merging makes dev match what is deployed.

Database isolation

  • RLS policies for experiment definitions, templates and versions, project template attachments, samples and sample links, recipe batches and proposals, and project membership. Admin, worker and write policies are added to the tables that already had RLS.
  • Enforcement on the nine experiment tables is controlled by DB_RLS_ENFORCEMENT (enforce / disable). project_members is always enforced, together with its new write grants.
  • A trigger blocks non-admin changes to an experiment's owner, project, equipment, template and request id, and limits equipment owners to execution-status fields.

Database roles

  • User requests: api_client, carrying the caller's id, role and organization for the whole session.
  • Background jobs: a new api_worker role (not superuser, cannot bypass RLS, no DDL).
  • Owner credentials are used only by migrations, which now run as a separate Job (python -m db_migrations, geddes/k8s/09-db-migrate-job.yaml).

Fixes

  • Template visibility is checked when creating, forking, attaching, viewing versions, using processes and seeding workflow examples.
  • Experiment creation writes its initial proposal status in the same transaction. A retry with the same client_request_id returns the original experiment; a reused key with a different payload returns 409.
  • Proposal generation is a durable job queue with leases, bounded retries and results saved atomically with completion, plus a sweeper in each API worker.
  • init_db.sql now works on a fresh database; three stale security tests are aligned with the project-workflow change in 004d902.
  • Pytest points database connections at a dead port unless DT_ALLOW_LIVE_DB_TESTS=1, so unit tests can't reach a port-forwarded production database.

Operations docs

  • api/ROW_LEVEL_SECURITY.md: access model, rollout steps and tested rollback order.
  • geddes/k8s/02-api.yaml: deployed image and settings, plus a description of the DB3 pilot layer on the live Deployment.
  • geddes/k8s/GLANCE_INGESTION_OPERATIONS.md: how GLANCE run 12822 was deferred, and how to defer a run in future.

Production state (2026-09-24/25)

  • The dt-api Deployment and dt-api-retrain CronJob run dt-api:rls-b02e9ac, with DB_RLS_ENFORCEMENT=enforce and DB_MIGRATE_ON_STARTUP=false.
  • api_worker is provisioned. The owner credentials were moved from dt-api-secret to dt-db-migration-secret.
  • GLANCE ingestion is unblocked. It had failed since 2026-09-04 on tool 3 run 12822 (11 h, about 97.6% idle logging). The run was deferred, not deleted, and polling has resumed.

Test plan

  • Unit tests: 143 passed, 0 failed.
  • RLS integration suite (api/tests/test_row_level_security_pg.py): 36/36 on a disposable Postgres 15 with enforcement on. With enforcement off, only the isolation-only assertions fail.
  • Rollback: the previous dev code works against the migrated database while enforcement is off.
  • Production smoke test: every read endpoint as all 17 active users (423 requests) was identical to the pre-deploy baseline after deploying, after enforcing, and after removing owner credentials.
  • Production isolation: per-role row counts, plus write probes in transactions that were rolled back. A PI can create a project and an experiment; a researcher writing into a PI's project and a PI claiming someone else's project are both rejected.
  • The 02:00 UTC retrain ran as api_worker and succeeded.
  • Reviewed by a second agent (Codex) over 9 rounds; all findings resolved, final verdict approve.

🤖 Generated with Claude Code

- Add RLS policies for experiment definitions, templates and versions,
  project workflow attachments, samples and sample links, recipe batches
  and proposals, and project membership, plus admin, worker and write
  policies on the tables that already enforced RLS. Enforcement for the new
  tables is switched by DB_RLS_ENFORCEMENT so it can be rolled out and
  rolled back separately from the deploy.
- Split database access into user-scoped api_client connections (identity
  set at session scope), a least-privileged api_worker role for background
  jobs, and owner credentials used only for migrations (db_migrations).
- Check template visibility when creating, forking or reading versions.
- Create experiments atomically with their initial proposal status, and
  return the existing experiment for a repeated client_request_id.
- Replace the fire-and-forget proposal thread with a durable job queue:
  leases, bounded retries, age limit, results committed with completion
  only while the claim is held, and a sweeper in each API worker.
- Keep unit tests away from live databases by default.
- Add a runbook (api/ROW_LEVEL_SECURITY.md) and a migration Job manifest.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Enforce RLS on project_members as soon as its write grants exist,
  independent of the experiment rollout switch, and let project readers see
  membership so open-project member counts still work.
- Only let a PI enrol themselves on a project created in the same
  transaction (ingestion-seeded projects cannot be claimed).
- Block non-admin changes to an experiment's owner, project, equipment,
  template or request id (trigger; foreign-key actions exempt).
- Require readable templates for experiment and project attachment writes;
  run process endpoints as the caller and check step templates.
- Fail closed in app_detach_experiment_type for NULL owners.
- Reject a reused client_request_id with a different payload (409); scope
  frontend keys to the generated recipe or unchanged form.
- Keep the migration ledger owner-only; bound migration lock waits and build
  new indexes outside the ALTER TABLE lock.
- Force pytest onto a dead database port unless explicitly opted in.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Apply the template visibility rule to direct project attachments,
  process step joins and workflow example seeding.
- Resolve a repeated client_request_id before validating against the
  current template, so retries survive template edits.
- Resend the exact stored save request from the optimization page.
- Limit facility managers' experiment updates to execution status columns.
- Document atomic standalone application and always-enforced membership.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Only refresh a workflow example template the caller owns (or as admin),
  independent of row-level security.
- Record an interrupted follow-up generation as failed instead of treating
  it as an initial proposal job.
- Use the configured migration lock timeout for the RLS migration.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Create or update api_client with \gexec: psql does not substitute
  variables inside dollar-quoted DO blocks, so fresh installs failed.
- Make init_db.sql policies re-runnable.
- Align three security tests with the project workflow change that made
  project equipment optional and allowed multi-equipment projects.
- Document the tested rollback order for older API images.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Deployed dt-api:rls-b02e9ac to the dt-api Deployment and the dt-api-retrain
CronJob, and enabled DB_RLS_ENFORCEMENT=enforce on the Deployment.

Note: the live dt-api Deployment also carries the DB3 pilot ConfigMap and
uploads volume (geddes/k8s/pilots/db3), which 02-api.yaml does not include;
apply changes with `kubectl set image`/`set env`, not `kubectl apply` of
this file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Record DB_MIGRATE_ON_STARTUP=false (migrations now run as the
  dt-db-migrate Job; the API no longer has owner credentials).
- Match the live env list (the two entries missing live were applied;
  both were functional no-ops).
- Document the DB3 pilot layer that pilots/db3/deploy.sh adds on top of
  this file, and how to change the Deployment without removing it.
  `kubectl diff -f` now shows only that layer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Describe dt-db-migrate as the migration mechanism and require it to
  complete before each rollout; enforcement is switched through
  dt-db-migration-secret.
- Note that the Deployment's DB_RLS_ENFORCEMENT is informational while
  startup migrations are off.
- Replace the pilot restore advice (deploy.sh rejects non-pilot images)
  with a configuration-only patch that keeps the current image.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Tool 3 run 12822 (11 h, 97.6% idle logging) exceeded the connector's value
limit and blocked ingestion from 2026-09-04. Document why it was deferred
rather than ingested whole, the checks made, the procedure used (temporary
initial_run_id floor, restored afterwards) and how to backfill it later.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Describe the poll page as including overlap rows, and the initial_run_id
  floor as applying to normal polling with backfill settings unset.
- Replace the run-count estimate with the observed backlog bounds.
- Distinguish computed trace means from persisted features; note the
  backfill needs a value limit that fits the run.
- Record the 05:40 poll after restoring the floor.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign in to join this conversation on GitHub.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant